Visible light image and infrared image fusion processing system and fusion method

ABSTRACT

The invention relates to a visible light image and infrared image fusion processing system and a fusion method. The fusion processing system comprises an image acquisition module, an image fusion module and an image display module, wherein the image fusion module is connected with the image acquisition module and the image display module. By adoption of the fusion method, the fusion ratio of a visible light image to an infrared image can be adjusted according to requirements, and detailed images are filtered and compared and then enhanced, so that detail information of a fused image is improved, and noise interference is avoided. Furthermore, fusion weights of the visible light image and the infrared image and the detail enhancement degree can be flexibly controlled through external parameter adjustment, and thus various display requirements are met.

BACKGROUND OF THE INVENTION Technical Field

The invention belongs to the field of digital image processing, andparticularly relates to a visible light image and infrared image fusionprocessing system and fusion method.

Description of Related Art

In the field of digital image processing, imaging principles of variousimage sensors are different, core devices used for photoelectricconversion respond to wavelengths within different ranges, and a singleimage sensor cannot meet various imaging requirements.

BRIEF SUMMARY OF THE INVENTION

To overcome the imaging defects of a single image sensor, the inventionprovides a visible light image and infrared image fusion processingsystem and a fusion method, which can meet different imagingrequirements for different sensors under different conditions; detailedimages which are denoised and filtered are enhanced by screening outextreme values, so that detail defects and loud noise of a traditionalfusion system are avoided. By adoption of the visible light image andinfrared image fusion processing system and the fusion method, thefusion ratio of visible light images to infrared images can be adjustedaccording to requirements, images are enhanced in detail, and thusdetailed information of fused images is improved.

The technical solution of the invention is as follows:

A visible light image and infrared image fusion processing systemcomprises an image acquisition module, an image fusion module and animage display module, wherein the image fusion module is connected withthe image acquisition module and the image display module respectively;

The image acquisition module comprises an infrared image acquisitiondevice and a visible light image acquisition device, and the imagefusion module is used for fusion processing of an infrared image and avisible light image, so that a fused image is obtained; the fused imageis transmitted to a display device to be displayed through the imagedisplay module.

The lenses of the infrared image acquisition device and the visiblelight image acquisition device are mounted at the same position, andoptical axes of the lenses are in the same direction and in parallel;the infrared image acquisition device and the visible light imageacquisition device need to synchronously output video images frame byframe, and the field angle range is registered according to theresolution, so that the images have the same scene area; the area whereregistered images are selected is preset, and thus image registrationcan be achieved without complex calculation.

The fusion method based on the visible light image and infrared imagefusion processing system comprises the following steps:

Step 1: the format of the selected target area of a visible light imageis transformed, so that the color image is converted into a grayscaleimage or only a luminance component image of the color image isselected;

Step 2: the visible grayscale image or the luminance component image islow-pass filtered, so that a low-frequency component image of thevisible light image is obtained; then the difference image data betweenthe non-filtered visible grayscale image and the visible low-frequencycomponent image is calculated, so that a visible high-frequencycomponent image is obtained;

Step 3: an infrared image is low-pass filtered, so that a low-frequencycomponent image of the infrared image is obtained; then the differenceimage data between a non-filtered infrared grayscale image and thelow-frequency component image of the infrared image is calculated, sothat a high-frequency component image of the infrared image is obtained;

Step 4: pseudo-color enhancement of the low-frequency component image ofthe infrared image is achieved through the table look-up method, and theluminance component in a pseudo-color image is extracted;

Step 5: the low-frequency component images are fused, specifically, theweights of the low-frequency luminance component of the infrared imageand the low-frequency component of the visible grayscale image aresummed, and the weight sum of each pixel is one;

For keeping the target focused by human eyes unaffected, during fusion,the weight value of each pixel is adaptively calculated according toscene information based on the following principles: for the firstprinciple, the value range of the luminance component image of theinfrared low-frequency image is [0, 255], and the value range of thelow-frequency component image of the visible light image is also [0,255]; for the second principle, when the focused scene mainly depends onthe infrared image, if the grayscale value of the luminance componentimage of the infrared low-frequency image is greater than that of thelow-frequency component image of the visible light image, the weightvalue of each pixel of the infrared image is one, otherwise, the weightvalue of each pixel of the infrared image is set according to parametersinput from the outside; for the third principle, when the focused scenemainly depends on the visible light image, if the grayscale value of theluminance component image of the infrared low-frequency image is greaterthan that of the low-frequency component image of the visible lightimage, the weight value of the infrared image is set according toparameters input from the outside, otherwise, the weight value of theinfrared image is zero; for the fourth principle, no matter whether thefocused scene mainly depends on the infrared image or the visible lightimage, the weight sum of the infrared image and the visible light imageis one all the time.

Step 6: the high-frequency component images are enhanced, specifically,the enhancement degree of the high-frequency component image of theinfrared image and the high-frequency component image of the visiblelight image is adjusted through control parameters;

Step 7: the enhanced high-frequency component image of the infraredimage and the enhanced visible high-frequency component image aresuperposed on the fused low-frequency component image obtained in Step5, and thus a fused luminance component image is obtained;

Step 8: the luminance component image in the infrared pseudo-color imageis replaced with fused luminance component image, and thus a final fusedimage is obtained.

Step 6 specifically comprises the following sub-steps:

Sub-step 6.1: for weakening noise of detailed images, the high-frequencyimages need to be bilaterally filtered before the detailed images areenhanced, and thus noise is removed;

Sub-step 6.2: the detail value of the infrared image and the detailvalue of the visible light image are compared; if the pixel values ofthe detailed images are both positive values, the greater one is used asthe final detail value; if the pixel values of the detailed images areboth negative values, the smaller one is used as the final detail value;else the greater absolute value is used as the final detail value, andthus a new detailed image is obtained;

Sub-step 6.3: the detailed image is enhanced through an externalenhancement parameter, so that an enhanced high-frequency componentimage is obtained.

The visible light image and infrared image fusion processing system andthe fusion method of the invention have the beneficial effects that thefusion ratio of the visible light image to the infrared image can beadjusted according to focusing requirements, and the detailed images arefiltered and compared and then enhanced, so that detail information ofthe fused image is improved, and noise interference is avoided.Furthermore, the fusion weights of the visible light image and theinfrared image and the detail enhancement degree can be flexiblycontrolled through external parameter adjustment, and thus variousdisplay requirements are met.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a schematic diagram of a system of the invention.

FIG. 2 is a flow diagram of an image fusion method of the invention.

DETAILED DESCRIPTION OF THE INVENTION

A further description of the invention is given with the accompanyingdrawings.

As is shown in FIG. 1, the present invention comprises an imageacquisition module, an image fusion module and an image display module,wherein the image fusion module is connected with the image acquisitionmodule and the image display module respectively; the image acquisitionmodule comprises an infrared image acquisition device and a visiblelight image acquisition device, and the image fusion module is used forfusion processing of an infrared image and a visible light image, sothat a fused image is obtained; the fused image is transmitted to adisplay device to be displayed through the image display module.

The image acquisition module is composed of the infrared imageacquisition device and the visible light image acquisition device,lenses of the infrared image acquisition device and the visible lightimage acquisition device must be mounted at the same position, andoptical axes of the lenses are in the same direction and in parallel;the different image acquisition devices need to synchronously outputvideo images frame by frame, and the field angle range is registeredaccording to the resolution, so that the images have the same scenearea; the area where registered images are selected can be preset, andthus image registration can be achieved without complex calculation. Theimage acquisition module transmits acquired infrared image data andvisible light image data to the image fusion processing module.

Image fusion of areas of interest of the selected infrared and visiblelight images is achieved by the image fusion module through softwareoperation, the specific realization process is shown in FIG. 2, and thefusion method comprises the following detailed steps:

Step 1: the format of the selected target area of a visible light imageis transformed, so that the color image is converted into a grayscaleimage or only a luminance component image of the color image isselected;

Step 2: the visible grayscale image or the luminance component image islow-pass filtered, so that a low-frequency component image of thevisible light image is obtained; then the difference image data betweenthe non-filtered visible grayscale image and the visible low-frequencycomponent image is calculated, so that a visible high-frequencycomponent image is obtained;

Step 3: an infrared image is low-pass filtered, so that a low-frequencycomponent image of the infrared image is obtained; then the differencebetween a non-filtered infrared grayscale image and the low-frequencycomponent image of the infrared image is calculated, so that ahigh-frequency component image of the infrared image is obtained;

Step 4: the pseudo-color reality augmentation of the low-frequencycomponent image of the infrared image is achieved through the tablelook-up method, and the luminance component in a pseudo-color image isextracted;

Step 5: the low-frequency component images are fused, specifically, theweights of the low-frequency luminance component of the infrared imageand the low-frequency component of the visible grayscale image aresummed, and the weight sum of each pixel is one;

For keeping the target focused by human eyes unaffected, during fusion,the weight value of each pixel is adaptively calculated according toscene information based on the following principles: for the firstprinciple, the value range of the luminance component image of theinfrared low-frequency image is [0, 255], and the value range of thelow-frequency component image of the visible light image is also [0,255]; for the second principle, when the focused scene mainly depends onthe infrared image, if the grayscale value of the luminance componentimage of the infrared low-frequency image is greater than that of thelow-frequency component image of the visible light image, the weightvalue of each pixel of the infrared image is one, otherwise, the weightvalue of each pixel of the infrared image is set according to parametersinput from the outside; for the third principle, when the focused scenemainly depends on the visible light image, if the grayscale value of theluminance component image of the infrared low-frequency image is greaterthan that of the low-frequency component image of the visible lightimage, the weight value of the infrared image is set according toparameters input from the outside, otherwise, the weight value of theinfrared image is zero; for the fourth principle, no matter whether thefocused scene mainly depends on the infrared image or the visible lightimage, the weight sum of the infrared image and the visible light imageis one all the time.

Step 6: the high-frequency component images are enhanced, specifically,the enhancement degree of the high-frequency component image of theinfrared image and the high-frequency component image of the visiblelight image is adjusted through control parameters, and this stepspecifically comprises the following sub-steps:

Sub-step 6.1: for weakening noise of detailed images, the high-frequencyimages need to be bilaterally filtered before the detailed images areenhanced, and thus noise is removed;

Sub-step 6.2: the detail value of the infrared image and the detailvalue of the visible light image are compared; if the pixel values ofthe detailed images are both positive values, the greater one is used asthe final detail value; if the pixel values of the detailed images areboth negative values, the smaller one is used as the final detail value;else the greater absolute value is used as the final detail value, andthus a new detailed image is obtained;

Sub-step 6.3: the detailed image is enhanced through an externalenhancement parameter, so that an enhanced high-frequency componentimage is obtained.

Step 7: the enhanced high-frequency component image of the infraredimage and the enhanced visible high-frequency component image aresuperposed on the fused low-frequency component image obtained in Step5, and thus a fused luminance component image is obtained;

Step 8: the luminance component in the infrared pseudo-color image isreplaced with fused luminance component image, and thus a final fusedimage is obtained.

1. A visible light image and infrared image fusion processing system, comprising an image acquisition module, an image fusion module and an image display module, wherein: the image fusion module is connected with the image acquisition module and the image display module respectively; and the image acquisition module comprises an infrared image acquisition device and a visible light image acquisition device, and the image fusion module is used for fusion processing of an infrared image and a visible light image to obtain a fused image; the image display module transmits the fused image to a display device for show.
 2. The visible light image and infrared image fusion processing system according to claim 1, wherein: lenses of the infrared image acquisition device and the visible light image acquisition device are mounted at the same position, and optical axes of the lenses are in the same direction and in parallel; the infrared image acquisition device and the visible light image acquisition device need to synchronously output video images frame by frame, and the field angle range is registered according to the resolution, so that the images have the same scene area; and the area where registered images are selected is preset, and thus image registration can be achieved without complex calculation.
 3. A fusion method based on the visible light image and infrared image fusion processing system according to claim 1, wherein the fusion method comprising the following steps: Step 1: transforming the selected target region of the visible light image into a format, converting the color image into a grayscale image or selecting only the luminance component image of the color image; Step 2: filtering at a low-pass the grayscale image or the luminance component of the visible light to obtain the low-frequency component image of the visible light image; and then performing the difference calculation between the non-filtered visible grayscale image and the visible low-frequency component image to obtain a visible high-frequency component image; Step 3: filtering at a low pass the infrared image, to obtain a low-frequency component image of the infrared image; then performing the difference calculation between the non-filtered infrared grayscale image and the low-frequency component image of the infrared image to obtain a high-frequency component image of the infrared image; Step 4: Using the look-up table method to realize the pseudo-color enhancement reality of the low frequency component of the infrared image and extracting the luminance component of the pseudo-color image; Step 5: Fusing the low-frequency components of the image, and performing sum of the weights of the low-frequency luminance component of the infrared image and the low-frequency component of the visible grayscale image, and the weight sum of each pixel is one; Step 6: enhancing the high-frequency component images, and adjusting the high-frequency component image of the visible light image and the enhancement degree of the high-frequency component image of the infrared image by control parameters; Step 7: superposing the enhanced high-frequency component image of the infrared image and the enhanced visible high-frequency component image on the fused low-frequency component image obtained in Step 5 to obtain a fused luminance component image; Step 8: replacing the luminance component in the infrared pseudo-color image with the fused luminance component image, to obtain the final fused image.
 4. The fusion method based on the visible light image and infrared image fusion processing system according to claim 3, wherein step 6 specifically comprises the following sub-steps: Sub-step 6.1: for weakening noise of detailed images, filtering bilaterally the high-frequency images before enhancing the detailed images to remove noise; Sub-step 6.2: comparing the detail value of the infrared image and the detail value of the visible light image; if the pixel values of the detailed images are both positive values, the greater one is used as the final detail value; if the pixel values of the detailed images are both negative values, the smaller one is used as the final detail value, and thus obtaining a new detailed image; Sub-step 6.3: enhancing the detailed image by an external enhancement parameter to obtain an enhanced high-frequency component image. 